Development notes: replacing fixed constants with data-derived parameters - #5
Draft
agulaya24 wants to merge 2 commits into
Draft
Development notes: replacing fixed constants with data-derived parameters#5agulaya24 wants to merge 2 commits into
agulaya24 wants to merge 2 commits into
Conversation
…rameters Records the August measurement work for the public repo. The through-line is removing magic numbers by letting the data determine its own measurement parameters, which became necessary on large dense personal corpora and on opening up organizational use cases. Covers, with numbers: a distance conversion that was wrong for both Chroma spaces and that our own architecture doc stated as fact; a faithfulness guard set below the noise floor of the encoder in use, so it could never fire; a deduplication step measured at 83% correct when two facts are the same and 23% when they are not; what that merge behaviour revealed about facts that disagree; and a 756-test suite that stayed green through all of it. Examples are abstract. No corpus content. States explicitly that the work is unmerged and that main still ships the previous behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qw1sMXN1yma2eCi8zZcdpd
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
An independent check found three statements in the note stronger than the evidence behind them: - The faithfulness guard section said random pairs have a minimum similarity of 0.358 and that the guard could therefore never fire. That was a sample minimum over 600 pairs. Across roughly eight million within-corpus pairs the true minimum is 0.276 and about 0.015% fall below 0.35. The guard is effectively unreachable for an on-topic English summary, which is what the evidence supports. - The 83% and 23% figures have wide confidence intervals. The direction holds (Fisher exact p = 0.001) but the magnitudes are not quotable, so the section now leads with the population figure instead. - Test count 756 to 769. Also records two limitations of our own method that cut against the conclusion: the labelling rubric instructed the labeller to be strict about calling pairs distinct, which biases ground truth toward finding the pipeline merge-happy, and the harness supplied one candidate neighbour where production supplies three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qw1sMXN1yma2eCi8zZcdpd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
docs/DEVELOPMENT_NOTES.md, a dated record of work in progress. One new file, no code changes.What it says
The through-line is removing magic numbers by letting the data determine its own measurement parameters. That became necessary when running on large, dense personal corpora and when opening up organizational use cases, where constants that looked reasonable at small scale turned out to sit in the wrong part of the distribution. Nothing errored; the numbers stopped meaning what their names implied.
Most of the note is a defect report on our own code, published because a project that reports only what worked is not one whose measurements anyone should trust.
Covered, with numbers:
docs/core/ARCHITECTURE.mdstated as fact. A doc that states a formula is where the next contributor learns to reimplement it.Scope and safety
mainstill ships the previous behaviour, so nothing here reads as a released capability.Draft while the described work is still in progress.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Qw1sMXN1yma2eCi8zZcdpd